寻找特定任务说明的YouTube用户可能会花费很长时间浏览内容，以寻找与他们需求相匹配的正确视频。创建视觉摘要（视频的删节版本）为观众提供了快速概述，并大大减少了搜索时间。在这项工作中，我们专注于总结教学视频，这​​是视频摘要的探索领域。与通用视频相比，可以将教学视频解析为语义上有意义的细分，这些细分与所示任务的重要步骤相对应。现有的视频摘要数据集依靠手动框架级注释，使其主观且大小有限。为了克服这一点，我们首先通过利用两个关键假设来自动为教学视频语料库生成伪摘要：（i）相关步骤可能会出现在相同任务（任务相关性）的多个视频中，并且（ii）它们更重要。可能由示威者口头描述（跨模式显着）。我们提出了一个教学视频摘要网络，该网络结合了上下文感知的时间视频编码器和段评分变压器。使用伪摘要作为弱监督，我们的网络为仅给出视频和转录语音的教学视频构建了视觉摘要。为了评估我们的模型，我们通过刮擦包含视频演示的Wikihow文章和步骤的视觉描绘，从而收集了高质量的测试集，即Wikihow摘要，从而使我们能够获得地面真实性摘要。我们的表现优于几个基线和这个新基准的最先进的视频摘要模型。

我们向多人3D运动轨迹预测提出了一种新颖的框架。我们的主要观察是，人类的行动和行为可能高度依赖于其他人。因此，不是以隔离预测每个人类姿势轨迹，我们引入了一种多范围变压器模型，该模型包含用于各个运动的局部运动和用于社交交互的全局范围编码器。然后，通过将相应的姿势作为查询来参加本地和全球范围编码器特征，对变压器解码器对每个人进行预测。我们的模型不仅优于长期3D运动预测的最先进的方法，而且还产生了不同的社交互动。更有趣的是，我们的模型甚至可以通过自动将人分为不同的交互组来同时预测15人运动。具有代码的项目页面可在https://jiahunwang.github.io/mrt/处获得。

通用视频摘要是一种传播全部故事并具有最重要的场景的视频的销钉版本。然而，视频中场景的重要性通常是主观的，并且用户应该可以选择通过使用自然语言来定制摘要来指定对它们重要的内容。此外，用于全自动通用摘要的现有模型没有利用可用的语言模型，可以作为显着性的有效性。这项工作引入了剪辑 - 它，一个框架，用于解决通用和查询的视频摘要，通常在文献中单独接近。我们提出了一种语言引导的多模式变压器，该变压器学习基于它们相对于彼此的重要性以及与用户定义的查询（用于查询集中的摘要）或自动生成的密集视频字幕的关联（用于泛型视频摘要）。我们的模型可以通过培训延伸到无监督的环境，而没有地理监督。我们以标准视频摘要数据集（TVSUM和SUMME）和查询视频摘要数据集（QFVS）在标准视频摘要数据集（TVSUM和SUMPE）上的重大边际而先前的工作。特别是，我们在转移环境中取得了大量的改进，证明了我们的方法的强大泛化能力。

We consider the task of text generation in language models with constraints specified in natural language. To this end, we first create a challenging benchmark Cognac that provides as input to the model a topic with example text, along with a constraint on text to be avoided. Unlike prior work, our benchmark contains knowledge-intensive constraints sourced from databases like Wordnet and Wikidata, which allows for straightforward evaluation while striking a balance between broad attribute-level and narrow lexical-level controls. We find that even state-of-the-art language models like GPT-3 fail often on this task, and propose a solution to leverage a language model's own internal knowledge to guide generation. Our method, called CognacGen, first queries the language model to generate guidance terms for a specified topic or constraint, and uses the guidance to modify the model's token generation probabilities. We propose three forms of guidance (binary verifier, top-k tokens, textual example), and employ prefix-tuning approaches to distill the guidance to tackle diverse natural language constraints. Through extensive empirical evaluations, we demonstrate that CognacGen can successfully generalize to unseen instructions and outperform competitive baselines in generating constraint conforming text.

Building segmentation in high-resolution InSAR images is a challenging task that can be useful for large-scale surveillance. Although complex-valued deep learning networks perform better than their real-valued counterparts for complex-valued SAR data, phase information is not retained throughout the network, which causes a loss of information. This paper proposes a Fully Complex-valued, Fully Convolutional Multi-feature Fusion Network(FC2MFN) for building semantic segmentation on InSAR images using a novel, fully complex-valued learning scheme. The network learns multi-scale features, performs multi-feature fusion, and has a complex-valued output. For the particularity of complex-valued InSAR data, a new complex-valued pooling layer is proposed that compares complex numbers considering their magnitude and phase. This helps the network retain the phase information even through the pooling layer. Experimental results on the simulated InSAR dataset show that FC2MFN achieves better results compared to other state-of-the-art methods in terms of segmentation performance and model complexity.

Fine-tuning pre-trained language models (PLMs) achieves impressive performance on a range of downstream tasks, and their sizes have consequently been getting bigger. Since a different copy of the model is required for each task, this paradigm is infeasible for storage-constrained edge devices like mobile phones. In this paper, we propose SPARTAN, a parameter efficient (PE) and computationally fast architecture for edge devices that adds hierarchically organized sparse memory after each Transformer layer. SPARTAN freezes the PLM parameters and fine-tunes only its memory, thus significantly reducing storage costs by re-using the PLM backbone for different tasks. SPARTAN contains two levels of memory, with only a sparse subset of parents being chosen in the first level for each input, and children cells corresponding to those parents being used to compute an output representation. This sparsity combined with other architecture optimizations improves SPARTAN's throughput by over 90% during inference on a Raspberry Pi 4 when compared to PE baselines (adapters) while also outperforming the latter by 0.1 points on the GLUE benchmark. Further, it can be trained 34% faster in a few-shot setting, while performing within 0.9 points of adapters. Qualitative analysis shows that different parent cells in SPARTAN specialize in different topics, thus dividing responsibility efficiently.

While large language models (LLMs) have demonstrated impressive capabilities across tasks in language understanding and interactive decision making, their abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g. action plan generation) have primarily been studied as separate topics. In this paper, we explore the use of LLMs to generate both reasoning traces and task-specific actions in an interleaved manner, allowing for greater synergy between the two: reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it to interface with external sources, such as knowledge bases or environments, to gather additional information. We apply our approach, named ReAct, to a diverse set of language and decision making tasks and demonstrate its effectiveness over state-of-the-art baselines, as well as improved human interpretability and trustworthiness over methods without reasoning or acting components. Concretely, on question answering (HotpotQA) and fact verification (Fever), ReAct overcomes issues of hallucination and error propagation prevalent in chain-of-thought reasoning by interacting with a simple Wikipedia API, and generates human-like task-solving trajectories that are more interpretable than baselines without reasoning traces. On two interactive decision making benchmarks (ALFWorld and WebShop), ReAct outperforms imitation and reinforcement learning methods by an absolute success rate of 34% and 10% respectively, while being prompted with only one or two in-context examples. Project site with code: https://react-lm.github.io

接近周期性的模式（NPP）在人造场景中无处不在，由瓷砖图案组成，其外观差异是由照明，缺陷或设计元素引起的。良好的NPP表示对许多应用程序有用，包括图像完成，分割和几何重新映射。但是代表NPP是具有挑战性的，因为它需要保持全球一致性（瓷砖图案布局），同时保留局部变化（外观差异）。使用大型数据集或单图像优化斗争在一般场景上训练的方法以满足这些约束，而明确模型周期性的方法对周期性检测错误并不强大。为了应对这些挑战，我们使用基于坐标的MLP学习具有单图像优化的神经隐式表示。我们设计一个输入功能翘曲模块和周期性指导的补丁损失，以处理全球一致性和局部变化。为了进一步提高鲁棒性，我们引入了一个周期性建议模块，以在我们的管道中搜索和使用多个候选周期。我们在单个和多平面场景上展示了我们方法对500多个建筑物，架子，壁纸，地面和蒙德里安图案的有效性。

骨肉瘤是最常见的原发性骨癌，其标准治疗包括术前化疗，然后切除。化学疗法反应用于预测患者的预后和进一步治疗。坏死在切除标本上的组织学幻灯片通常评估了坏死比定义为坏死肿瘤与总体肿瘤之比。已知坏死比> = 90％的患者的预后更好。多个载玻片对坏死比的手动微观综述是半定量性的，并且可能具有观察者间和观察者间的变异性。我们提出了一种基于目标和可再现的深度学习方法，以估计坏死比，并从扫描的苏木精和曙红全幻灯片图像预测结果。我们以3134个WSI的速度收集了103例骨肉瘤病例，以训练我们的深度学习模型，验证坏死比评估并评估结果预测。我们训练了深层多磁化网络，以分割多个组织亚型，包括生存的肿瘤和像素级中的坏死肿瘤，并计算来自多个WSI的病例级坏死比。我们显示了通过分割模型估算的坏死比，高度与由专家手动评估的病理报告中的坏死比高度相关，其中IV级的平均绝对差异（100％），III（> = 90％）和II（> = 50％和<50％和< 90％）坏死反应分别为4.4％，4.5％和17.8％。我们成功地对患者进行了分层，以预测P = 10^-6的总生存率，而P = 0.012的无进展生存率。我们没有可变性的可重现方法使我们能够调整截止阈值，特别是用于模型和数据集的截止阈值，为OS的80％，PFS为60％。我们的研究表明，深度学习可以支持病理学家作为一种客观的工具，可以分析组织学中骨肉瘤，以评估治疗反应并预测患者结果。

在交互式环境中，现有的基础语言基准要么缺乏现实世界的语言元素，要么由于人类参与数据收集或反馈信号而难以扩展。为了弥合这一差距，我们开发了网络商店 - 一个模拟的电子商务网站环境，拥有11.18亿美元的现实世界中的产品和12,087美元的人群文本说明。给定指定产品需求的文本指令，代理需要导航多种类型的网页并发布各种操作以查找，自定义和购买项目。 WebShop为语言基础提供了一些挑战，包括了解构图说明，查询（重新）表述，理解和对网页中的嘈杂文本进行操作以及执行战略探索。我们为这项任务收集了超过1,600美元的人类示范，并使用强化学习，模仿学习以及预训练的图像和语言模型来训练和评估各种代理商。我们的最佳模型达到了任务成功率$ 29 \％$，它优于基于规则的启发式方法（$ 9.6 \％$），但远低于人类专家绩效（$ 59 \％$）。我们还分析了代理和人类轨迹，并消融各种模型组件，以提供有关具有更强语言理解和决策能力的未来代理人的见解。最后，我们表明，在Amazon.com上进行评估时，在网络商店进行培训的代理商展示了非平凡的SIM转移转移，这表明网络商店在开发可以在野外运行的实用基于网络的代理商中的潜在价值。